Hello

Did you come prepared?

Learning Goals

1. Apply and interpret OLS for Linear Models
2. Identify and address problems with Linear Models

Linear Model

1. Used for prediction and inference when response is quantitative.
2. Applied when the relation bentwen the response and predictor(s) is assumed to be close to linear.
3. A linear model with one predictor is referred to as the Simple Linear model and that with more than one predictors is called Multiple Linear Model.

Simple Linear Model

Simple Linear Model

• A perfect linear relationship is unrealistic for any natural process.
• So, it is not possible to predict the exact value of \(y\) just by knowing \(x\).
• Ex: Family income (\(x\)) and financial support (\(y\)) to a student by a college.
• This doesn’t mean one cannot make a reasonably good estimate of \(y\) using \(x\).

Simple Linear Model

The relationship between \(x\) and \(y\) can be modeled as a straight line with some error:

\(y = b_0 + b_1x + \epsilon\)

\(b_0\) is the intercept and \(b_1\) is the slope of the line. Error is represented by \(\epsilon\)

Possums

Possums

Possums

The equation of the line he have is:

\(\hat{y} = 42.7 + 0.573x\)

For possums of total length 85 cm, we estimate the average head length to be:

\(\hat{y} = 42.7 + 0.573*(85) = 91.405\)

Residuals

Residual

Formally, we refer to residuals as :

for the \(i^{th}\) observation, residual \(e_i= y_i - \hat{y}_i\)

DIY-1 {2 mins}

Use the linear model \(\hat{y} = 41 + 0.59x\) to compute the residual for the observation (76.0, 85.1).

DIY-2

If a model underestimates an observation, will the residual be positive or negative?

Residuals

Least Squares

Provides and Objective measure of finding the best line

A line that has the smallest residuals

\(e_1^2 + e_2^2 + ... + e_n^2\)

Interpretation

For a model:

\(\hat{y} = \beta_0 + \beta_1 x\)

The slope(\(\beta_1\)) describes the estimated difference in the predicted average outcome of \(y\) if the predictor variable \(x\) happened to be one unit larger.

The intercept describes the average outcome of \(y\) if \(x = 0\)

Extrapolation

Extrapolation

lm(gift_aid ~ family_income,
1    data = elmhurst) |>
2    broom::augment() |>
    ggplot(aes(x = family_income, gift_aid)) +
    geom_point(colour = "steelblue") +
    geom_smooth(method = "lm",se = F,linetype =2)+
    geom_segment(aes(xend = family_income, yend=.fitted),
    colour = "red", alpha = 0.5) +
    labs(
        x = "Family Income in 1000 USD",
        y = "Gift Aid in 1000 USD"
    ) +
        theme_minimal()

Extrapolation

lm(gift_aid ~ family_income,
    data = elmhurst) |>                                   
    broom::tidy() 

# A tibble: 2 × 5
  term          estimate std.error statistic  p.value
  <chr>            <dbl>     <dbl>     <dbl>    <dbl>
1 (Intercept)    24.3       1.29       18.8  8.28e-24
2 family_income  -0.0431    0.0108     -3.98 2.29e- 4

extrapolation

What will be the gift aid received by a student whose Family Income is $1Million?

\(GiftAid = 24.3 - 0.04318FamilyIncome\)

\(GiftAid = 24.3 - 0.0431 * 1000 = - 18.8\)

Does this meant the student will be penalised $18800?

Model fit

\(R^2\) is used to describe the strength of the model fit

It is the amount of variance in response explained by the model

\(\frac{Var(Gift Aid) - Var(Residuals)}{Var(Gift Aid)}\)

Outliers -Intuitive

O1-From IMS 2e

Outliers -Intuitive

O2-From IMS 2e

Outliers -Intuitive

1. Analyse with and without the outliers. Are the results different? Think why?
2. Present the differneces for discussion.
3. Do not remove these without good reason.

Multiple Linear Model

When many variables are associated with the response at once

Why do we need Multiple Linear Model

Why can’t I run several simple liner models?

1. How would you make a single prediction form many simple linear model?
2. Each simple linear model will ignore other factors that are associated with the response.

Multiple Linear Model

lm(head_l ~ total_l + sex + age , data = possum) |>
    broom::tidy() |>
    kableExtra::kable()

Are all the predictors zero?

F-statistic to the rescue

\(F = \frac{(TSS -RSS)/p}{RSS/(n-p-1)}\)

But what is large enough?

Variable selection

1. Forward Selection
2. Backward Selection

Adjusted \(R^2\)

\(R_{adj} ^2 = 1 - \frac{s_{residual}^2/(n-k-1)}{s_{outcome}^2/(n-1)}\)

Why not just use \(R^2\)?

“The adjusted R-squared adjusts for the number of terms in the model. Importantly, its value increases only when the new term improves the model fit more than expected by chance alone. The adjusted R-squared value actually decreases when the term doesn’t improve the model fit by a sufficient amount.”

Prediction

1. \(\beta_0, \beta_1,...\beta_p\) are estimates of population parameters. It is related ot the reducible error we talked about in the Bias-variance tradeoff. To address this uncertainty we use confidence Intervals. This is how close \(\hat{Y}\) is to \(f(X)\).

2. Even if we got perfect estimates of the paramenters, we have to deal with the irreducible error () that is hidden in every realization of \(Y\). To indicate this we use prediction intervals. This is how much \(Y\) vary from \(\hat{Y}\).

Exercise

• Use the duke_forest data from {openintro}
• Your goal is to model the price of the house
• Begin by reading the data documentation
• Carry out necessary exploratory analyses
• come up with a model
• Implement it
• Carry out diagnostics
• Tune the model if needed

Readings

Intro to modern statistics Chapters 7 and 8.